Adaptive Knowledge Driven Regularization for Deep Neural Networks

In many real-world applications, the amount of data available for training is often limited, and thus inductive bias auxiliary knowledge are much needed regularizing model training. One popular regularization method to impose prior distribution assumptions on parameters, recent works also attempt regularize by integrating external into specific neurons. However, existing methods did not take account interaction between connected neuron pairs, which invaluable internal adaptive better representation learning as progresses. this paper, we explicitly neurons, propose an driven method, CORR-Reg. The key idea CORR-Reg give a higher significance weight connections more correlated pairs. weights adaptively identify important input neurons each neuron. Instead connection parameters with static strength such decay, imposes weaker significant connections. As consequence, attend informative features learn diversified discriminative representation. We derive Bayesian inference framework novel optimization algorithm Lagrange multiplier Stochastic Gradient Descent. Extensive evaluations diverse benchmark datasets neural network structures show that achieves improvement over state-of-the-art methods.